Documentation
Looking at the data
We plot the data and can see that there is no obvious large difference between the debt versions.
d.both_completed %>%
ggplot(aes(high_debt_version, fill = documentation)) +
geom_bar(position = "fill") +
scale_y_reverse() +
scale_fill_manual("legend", values = c("Correct" = "green", "None" = "yellow", "Incorrect" = "red"), guide = guide_legend(reverse = TRUE))
Initial model
The type of the outcome is adjacent categories and is modeled as a cumulative family. TODO FIX!
We include high_debt_verison as well as a varying intercept for each individual in our initial model.
Selecting Priors
We iterate over the model until we have sane priors
documentation0.with <- extendable_model(
base_name = "documentation0",
base_formula = "documentation ~ 1 + high_debt_version + (1 | session)",
base_priors = c(
prior(normal(0, 0.9), class = "b"),
prior(normal(0, 0.9), class = "Intercept"),
prior(exponential(1), class = "sd", dpar = "muIncorrect"),
prior(exponential(1), class = "sd", dpar = "muNone")
),
family = categorical(),
data = d.both_completed,
)
# Default priors:
prior_summary(documentation0.with(only_priors= TRUE)) prior class coef group resp dpar nlpar bound source
(flat) b muIncorrect default
(flat) b high_debt_versionfalse muIncorrect (vectorized)
(flat) b muNone default
(flat) b high_debt_versionfalse muNone (vectorized)
student_t(3, 0, 2.5) Intercept muIncorrect default
student_t(3, 0, 2.5) Intercept muNone default
student_t(3, 0, 2.5) sd muIncorrect default
student_t(3, 0, 2.5) sd muNone default
student_t(3, 0, 2.5) sd session muIncorrect (vectorized)
student_t(3, 0, 2.5) sd Intercept session muIncorrect (vectorized)
student_t(3, 0, 2.5) sd session muNone (vectorized)
student_t(3, 0, 2.5) sd Intercept session muNone (vectorized)# Our priors:
prior_summary(documentation0.with(sample_prior = "only"))Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category. prior class coef group resp dpar nlpar bound source
normal(0, 0.9) b muIncorrect default
normal(0, 0.9) b high_debt_versionfalse muIncorrect (vectorized)
normal(0, 0.9) b muNone default
normal(0, 0.9) b high_debt_versionfalse muNone (vectorized)
normal(0, 0.9) Intercept muIncorrect default
normal(0, 0.9) Intercept muNone default
exponential(1) sd muIncorrect user
exponential(1) sd muNone user
exponential(1) sd session muIncorrect (vectorized)
exponential(1) sd Intercept session muIncorrect (vectorized)
exponential(1) sd session muNone (vectorized)
exponential(1) sd Intercept session muNone (vectorized)# Prior predictive check
pp_check(documentation0.with(sample_prior = "only"), nsamples = 200, type = "bars")Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
Model fit
We check the posterior distribution and can see that the model seems to have been able to fit the data well
# Posterior predictive check
pp_check(documentation0.with(), nsamples = 100, type = "bars")Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.
summary(documentation0.with())Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category. Family: categorical
Links: muIncorrect = logit; muNone = logit
Formula: documentation ~ 1 + high_debt_version + (1 | session)
Data: as.data.frame(data) (Number of observations: 44)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~session (Number of levels: 22)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept) 1.32 0.97 0.06 3.56 1.01 675 1609
sd(muNone_Intercept) 1.51 1.07 0.04 3.99 1.01 496 1177
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept 0.20 0.63 -1.13 1.33 1.00 3196 2478
muNone_Intercept 0.78 0.59 -0.45 1.91 1.00 2713 2323
muIncorrect_high_debt_versionfalse -0.07 0.65 -1.31 1.20 1.00 4366 2796
muNone_high_debt_versionfalse 0.30 0.62 -0.96 1.55 1.00 4690 2912
Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).Model extenstions
We use loo to check some possible extensions on the model.
edlvl_prior <- c(
prior(dirichlet(2), class = "simo", coef = "moeducation_level1", dpar = "muIncorrect"),
prior(dirichlet(2), class = "simo", coef = "moeducation_level1", dpar = "muNone")
)
loo(
documentation0.with(),
documentation0.with("work_domain"),
documentation0.with("work_experience_programming.s"),
documentation0.with("work_experience_java.s"),
documentation0.with("education_field"),
documentation0.with("mo(education_level)", edlvl_prior),
documentation0.with("workplace_peer_review"),
documentation0.with("workplace_td_tracking"),
documentation0.with("workplace_pair_programming"),
documentation0.with("workplace_coding_standards"),
documentation0.with("scenario"),
documentation0.with("group"),
documentation0.with("order")
)Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with()'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("work_domain")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 10 observations with a pareto_k > 0.7 in model 'documentation0.with("work_experience_programming.s")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 8 observations with a pareto_k > 0.7 in model 'documentation0.with("work_experience_java.s")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 4 observations with a pareto_k > 0.7 in model 'documentation0.with("education_field")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation0.with("mo(education_level)", edlvl_prior)'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_peer_review")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_td_tracking")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 9 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_pair_programming")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_coding_standards")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 5 observations with a pareto_k > 0.7 in model 'documentation0.with("scenario")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 5 observations with a pareto_k > 0.7 in model 'documentation0.with("group")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("order")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Output of model 'documentation0.with()':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 25 56.8% 776
(0.5, 0.7] (ok) 17 38.6% 424
(0.7, 1] (bad) 2 4.5% 397
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("work_domain")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 26 59.1% 869
(0.5, 0.7] (ok) 12 27.3% 415
(0.7, 1] (bad) 6 13.6% 260
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("work_experience_programming.s")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 25 56.8% 999
(0.5, 0.7] (ok) 9 20.5% 423
(0.7, 1] (bad) 10 22.7% 141
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("work_experience_java.s")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 26 59.1% 955
(0.5, 0.7] (ok) 10 22.7% 260
(0.7, 1] (bad) 8 18.2% 127
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("education_field")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 26 59.1% 703
(0.5, 0.7] (ok) 14 31.8% 256
(0.7, 1] (bad) 4 9.1% 280
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("mo(education_level)", edlvl_prior)':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 26 59.1% 697
(0.5, 0.7] (ok) 15 34.1% 485
(0.7, 1] (bad) 3 6.8% 172
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("workplace_peer_review")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 26 59.1% 1070
(0.5, 0.7] (ok) 12 27.3% 473
(0.7, 1] (bad) 6 13.6% 176
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("workplace_td_tracking")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 21 47.7% 1031
(0.5, 0.7] (ok) 20 45.5% 214
(0.7, 1] (bad) 3 6.8% 262
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("workplace_pair_programming")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 23 52.3% 946
(0.5, 0.7] (ok) 12 27.3% 496
(0.7, 1] (bad) 9 20.5% 155
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("workplace_coding_standards")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 25 56.8% 580
(0.5, 0.7] (ok) 17 38.6% 318
(0.7, 1] (bad) 2 4.5% 529
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("scenario")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 21 47.7% 541
(0.5, 0.7] (ok) 18 40.9% 253
(0.7, 1] (bad) 5 11.4% 297
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("group")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 28 63.6% 464
(0.5, 0.7] (ok) 11 25.0% 271
(0.7, 1] (bad) 5 11.4% 252
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("order")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 21 47.7% 873
(0.5, 0.7] (ok) 17 38.6% 202
(0.7, 1] (bad) 6 13.6% 114
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Model comparisons:
elpd_diff se_diff
documentation0.with("workplace_peer_review") 0.0 0.0
documentation0.with("work_domain") -0.2 1.2
documentation0.with() -0.4 0.7
documentation0.with("workplace_td_tracking") -0.6 1.1
documentation0.with("workplace_coding_standards") -0.7 0.6
documentation0.with("education_field") -0.7 0.9
documentation0.with("group") -0.7 0.9
documentation0.with("workplace_pair_programming") -1.1 0.9
documentation0.with("mo(education_level)", edlvl_prior) -1.1 1.0
documentation0.with("work_experience_java.s") -1.2 1.1
documentation0.with("order") -1.3 1.5
documentation0.with("scenario") -1.5 0.9
documentation0.with("work_experience_programming.s") -1.5 1.1 We pick out some interesting variables and try combining them
loo(
documentation0.with(),
documentation0.with("work_domain"),
documentation0.with("workplace_peer_review"),
documentation0.with("workplace_td_tracking"),
documentation0.with(c("work_domain", "workplace_peer_review")),
documentation0.with(c("work_domain", "workplace_td_tracking")),
documentation0.with(c("workplace_peer_review", "workplace_td_tracking")),
documentation0.with(c("work_domain", "workplace_peer_review", "workplace_td_tracking"))
)Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with()'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("work_domain")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 6 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_peer_review")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation0.with("workplace_td_tracking")'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 2 observations with a pareto_k > 0.7 in model 'documentation0.with(c("work_domain", "workplace_peer_review"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 1 observations with a pareto_k > 0.7 in model 'documentation0.with(c("work_domain", "workplace_td_tracking"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 4 observations with a pareto_k > 0.7 in model 'documentation0.with(c("workplace_peer_review", "workplace_td_tracking"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 4 observations with a pareto_k > 0.7 in model 'documentation0.with(c("work_domain", "workplace_peer_review", "workplace_td_tracking"))'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Output of model 'documentation0.with()':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 25 56.8% 776
(0.5, 0.7] (ok) 17 38.6% 424
(0.7, 1] (bad) 2 4.5% 397
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("work_domain")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 26 59.1% 869
(0.5, 0.7] (ok) 12 27.3% 415
(0.7, 1] (bad) 6 13.6% 260
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("workplace_peer_review")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 26 59.1% 1070
(0.5, 0.7] (ok) 12 27.3% 473
(0.7, 1] (bad) 6 13.6% 176
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with("workplace_td_tracking")':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 21 47.7% 1031
(0.5, 0.7] (ok) 20 45.5% 214
(0.7, 1] (bad) 3 6.8% 262
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with(c("work_domain", "workplace_peer_review"))':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 28 63.6% 877
(0.5, 0.7] (ok) 14 31.8% 440
(0.7, 1] (bad) 2 4.5% 334
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with(c("work_domain", "workplace_td_tracking"))':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 28 63.6% 797
(0.5, 0.7] (ok) 15 34.1% 492
(0.7, 1] (bad) 1 2.3% 324
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with(c("workplace_peer_review", "workplace_td_tracking"))':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 25 56.8% 957
(0.5, 0.7] (ok) 15 34.1% 270
(0.7, 1] (bad) 4 9.1% 187
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation0.with(c("work_domain", "workplace_peer_review", "workplace_td_tracking"))':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 29 65.9% 1023
(0.5, 0.7] (ok) 11 25.0% 675
(0.7, 1] (bad) 4 9.1% 245
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Model comparisons:
elpd_diff se_diff
documentation0.with(c("work_domain", "workplace_peer_review", "workplace_td_tracking")) 0.0 0.0
documentation0.with(c("work_domain", "workplace_peer_review")) -0.4 0.7
documentation0.with(c("work_domain", "workplace_td_tracking")) -0.9 0.8
documentation0.with("workplace_peer_review") -1.0 1.0
documentation0.with(c("workplace_peer_review", "workplace_td_tracking")) -1.2 0.9
documentation0.with("work_domain") -1.2 0.8
documentation0.with() -1.4 1.0
documentation0.with("workplace_td_tracking") -1.5 0.9 Candidate models
We inspect some of our top performing models.
All models seems to have sampled nicely (rhat is ca 1 and fluffy plots) they also have about the same fit to the data end similar estimated for the high_debt_version beta parameter
documentation0 <- documentation0.with()Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.summary(documentation0) Family: categorical
Links: muIncorrect = logit; muNone = logit
Formula: documentation ~ 1 + high_debt_version + (1 | session)
Data: as.data.frame(data) (Number of observations: 44)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~session (Number of levels: 22)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept) 1.32 0.97 0.06 3.56 1.01 675 1609
sd(muNone_Intercept) 1.51 1.07 0.04 3.99 1.01 496 1177
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept 0.20 0.63 -1.13 1.33 1.00 3196 2478
muNone_Intercept 0.78 0.59 -0.45 1.91 1.00 2713 2323
muIncorrect_high_debt_versionfalse -0.07 0.65 -1.31 1.20 1.00 4366 2796
muNone_high_debt_versionfalse 0.30 0.62 -0.96 1.55 1.00 4690 2912
Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).ranef(documentation0)$session
, , muIncorrect_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.7470067 1.444398 -4.4761393 1.213727
6033d90a5af2c702367b3a96 -0.5784250 1.431037 -4.2015125 1.690868
6034fc165af2c702367b3a98 -0.5797671 1.491558 -4.4219490 1.817742
603500725af2c702367b3a99 1.4013453 1.661203 -0.5322208 5.435977
603f97625af2c702367b3a9d 1.3861995 1.727238 -0.5869947 5.880972
603fd5d95af2c702367b3a9e -0.5532681 1.499100 -4.2581675 1.768124
60409b7b5af2c702367b3a9f 1.3688257 1.716254 -0.6373885 5.764173
604b82b5a7718fbed181b336 -0.5383897 1.433520 -4.0766335 1.681776
6050c1bf856f36729d2e5218 1.3915309 1.708214 -0.5543257 5.785509
6050e1e7856f36729d2e5219 0.2889439 1.086331 -1.7661648 2.821988
6055fdc6856f36729d2e521b -0.5521420 1.400777 -4.0275050 1.594849
60589862856f36729d2e521f 0.3169461 1.105256 -1.8033922 2.901520
605a30a7856f36729d2e5221 -0.7225399 1.475082 -4.5018520 1.407789
605afa3a856f36729d2e5222 -0.7105838 1.450999 -4.5794740 1.340559
605c8bc6856f36729d2e5223 0.5291758 1.167988 -1.4395825 3.403833
605f3f2d856f36729d2e5224 -0.5483810 1.444796 -4.0999982 1.694556
605f46c3856f36729d2e5225 -0.5629205 1.471839 -4.1716535 1.856721
60605337856f36729d2e5226 0.3190511 1.092574 -1.6690627 2.885536
60609ae6856f36729d2e5228 0.5281284 1.125644 -1.3691678 3.230003
6061ce91856f36729d2e522e -0.7383476 1.468647 -4.5413172 1.343341
6061f106856f36729d2e5231 -0.5875646 1.473553 -4.4078757 1.661198
6068ea9f856f36729d2e523e -0.5777273 1.450281 -4.0351175 1.629520
, , muNone_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.05558621 1.118638 -2.4188793 2.3273868
6033d90a5af2c702367b3a96 1.24308232 1.708663 -0.9343491 5.7670788
6034fc165af2c702367b3a98 1.23147343 1.681031 -0.8807831 5.4403865
603500725af2c702367b3a99 -1.06045831 1.653527 -5.2565720 1.1949475
603f97625af2c702367b3a9d -1.02688063 1.688710 -5.3299650 1.3145548
603fd5d95af2c702367b3a9e 1.25282665 1.703018 -0.8742166 5.8370990
60409b7b5af2c702367b3a9f -1.03831960 1.667494 -5.3294795 1.0947363
604b82b5a7718fbed181b336 1.27507181 1.738235 -0.8283333 5.5911513
6050c1bf856f36729d2e5218 -0.99799157 1.650894 -5.1530970 1.3357060
6050e1e7856f36729d2e5219 -1.20729147 1.647126 -5.3844693 0.8409788
6055fdc6856f36729d2e521b 1.26583853 1.693720 -0.8129744 5.5921383
60589862856f36729d2e521f -1.17927492 1.644214 -5.3838805 0.9207417
605a30a7856f36729d2e5221 -0.04796134 1.094246 -2.3835817 2.2673958
605afa3a856f36729d2e5222 -0.03272103 1.087160 -2.3258822 2.2620213
605c8bc6856f36729d2e5223 0.13602894 1.157270 -2.2280525 2.6387448
605f3f2d856f36729d2e5224 1.27705082 1.745486 -0.8647861 5.6147463
605f46c3856f36729d2e5225 1.25360191 1.713079 -0.8028035 5.8153710
60605337856f36729d2e5226 -1.22352306 1.622208 -5.2776060 0.8247435
60609ae6856f36729d2e5228 0.12778528 1.151828 -2.1453255 2.6740225
6061ce91856f36729d2e522e -0.07847170 1.066914 -2.2713707 2.1473563
6061f106856f36729d2e5231 1.22218268 1.657472 -0.9209316 5.5727133
6068ea9f856f36729d2e523e 1.24733828 1.676535 -0.8875774 5.4576228plot(documentation0, ask = FALSE)
pp_check(documentation0, nsamples = 150, type= "bars")
documentation1 <- documentation0.with("workplace_peer_review")Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.summary(documentation1) Family: categorical
Links: muIncorrect = logit; muNone = logit
Formula: documentation ~ 1 + high_debt_version + (1 | session) + workplace_peer_review
Data: as.data.frame(data) (Number of observations: 44)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~session (Number of levels: 22)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept) 1.56 1.05 0.08 3.91 1.00 655 1558
sd(muNone_Intercept) 1.29 1.02 0.04 3.76 1.00 673 1650
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept 0.15 0.70 -1.31 1.49 1.00 3358 2562
muNone_Intercept 0.46 0.62 -0.80 1.72 1.00 3645 2789
muIncorrect_high_debt_versionfalse -0.09 0.66 -1.39 1.21 1.00 5105 2898
muIncorrect_workplace_peer_reviewfalse 0.04 0.74 -1.45 1.45 1.00 3753 2715
muNone_high_debt_versionfalse 0.30 0.62 -0.89 1.54 1.00 4307 2926
muNone_workplace_peer_reviewfalse 0.81 0.68 -0.55 2.10 1.00 3795 3050
Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).ranef(documentation1)$session
, , muIncorrect_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.7796731 1.555826 -4.7689282 1.635552
6033d90a5af2c702367b3a96 -0.6479978 1.581244 -4.5662600 1.843063
6034fc165af2c702367b3a98 -0.6813477 1.563078 -4.5821568 1.810310
603500725af2c702367b3a99 1.8350614 1.932535 -0.5654696 6.669127
603f97625af2c702367b3a9d 1.6808429 1.878981 -0.5818434 6.375892
603fd5d95af2c702367b3a9e -0.6639747 1.694110 -5.0239417 1.961679
60409b7b5af2c702367b3a9f 1.8157182 1.919123 -0.5260058 6.436726
604b82b5a7718fbed181b336 -0.6596849 1.642309 -4.7123495 1.956744
6050c1bf856f36729d2e5218 1.6666907 1.806904 -0.5922459 6.089604
6050e1e7856f36729d2e5219 0.3388241 1.179005 -1.8741260 2.946523
6055fdc6856f36729d2e521b -0.7868763 1.659068 -5.1004890 1.720580
60589862856f36729d2e521f 0.3627940 1.199089 -1.9899422 3.035509
605a30a7856f36729d2e5221 -0.9173599 1.566438 -4.9564925 1.319932
605afa3a856f36729d2e5222 -0.9515436 1.640601 -5.0961575 1.424776
605c8bc6856f36729d2e5223 0.5588737 1.233204 -1.6452240 3.379121
605f3f2d856f36729d2e5224 -0.7915956 1.687693 -5.2219490 1.652847
605f46c3856f36729d2e5225 -0.7949234 1.650671 -5.1311007 1.708503
60605337856f36729d2e5226 0.3235350 1.190782 -2.0464560 2.976035
60609ae6856f36729d2e5228 0.7649240 1.330889 -1.3516427 3.888544
6061ce91856f36729d2e522e -0.9571343 1.617258 -5.1593610 1.255922
6061f106856f36729d2e5231 -0.6561428 1.611278 -4.6556502 1.855628
6068ea9f856f36729d2e523e -0.6546241 1.637685 -4.6738263 1.969629
, , muNone_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.172876265 1.062679 -2.5141473 1.9823890
6033d90a5af2c702367b3a96 0.915285084 1.617227 -1.0179812 5.2208248
6034fc165af2c702367b3a98 0.904656562 1.539997 -1.0547840 5.0003953
603500725af2c702367b3a99 -0.887607319 1.516669 -4.8790233 1.1131557
603f97625af2c702367b3a9d -0.742023507 1.549659 -4.6616405 1.4136495
603fd5d95af2c702367b3a9e 0.905783634 1.575857 -1.0669485 4.9609140
60409b7b5af2c702367b3a9f -0.879110738 1.498599 -4.6937335 1.0903668
604b82b5a7718fbed181b336 0.920801477 1.564911 -1.0394685 5.1456170
6050c1bf856f36729d2e5218 -0.754435881 1.551021 -4.6688002 1.3040945
6050e1e7856f36729d2e5219 -0.942469933 1.530720 -5.0538467 0.9844937
6055fdc6856f36729d2e521b 1.104470666 1.557338 -0.7118958 5.1568310
60589862856f36729d2e521f -0.911326288 1.483166 -4.7708107 0.9305530
605a30a7856f36729d2e5221 0.032161305 1.010984 -2.1122598 2.2793915
605afa3a856f36729d2e5222 0.045549452 1.023600 -2.0464425 2.3758812
605c8bc6856f36729d2e5223 0.213486085 1.064093 -1.8354748 2.7442638
605f3f2d856f36729d2e5224 1.115918637 1.619768 -0.7700016 5.3101545
605f46c3856f36729d2e5225 1.080125952 1.578760 -0.7559829 5.0225058
60605337856f36729d2e5226 -0.935572646 1.542393 -4.9690372 0.9980503
60609ae6856f36729d2e5228 0.003317732 1.041522 -2.1978980 2.3569345
6061ce91856f36729d2e522e 0.011013405 1.004339 -2.0909662 2.1890850
6061f106856f36729d2e5231 0.890725275 1.513538 -1.0443862 4.9602315
6068ea9f856f36729d2e523e 0.908896633 1.558264 -1.0533830 5.1896355plot(documentation1, ask = FALSE)
pp_check(documentation1, nsamples = 150, type= "bars")
documentation2 <- documentation0.with(c("workplace_peer_review", "work_domain"))Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.summary(documentation2) Family: categorical
Links: muIncorrect = logit; muNone = logit
Formula: documentation ~ 1 + high_debt_version + (1 | session) + work_domain + workplace_peer_review
Data: as.data.frame(data) (Number of observations: 44)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~session (Number of levels: 22)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept) 1.23 0.97 0.02 3.57 1.00 919 1432
sd(muNone_Intercept) 1.19 0.97 0.04 3.54 1.00 789 1956
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept 0.14 0.70 -1.29 1.51 1.00 4141 2983
muNone_Intercept 0.54 0.68 -0.79 1.88 1.00 3904 2907
muIncorrect_high_debt_versionfalse -0.07 0.64 -1.33 1.18 1.00 5233 2898
muIncorrect_work_domainDevops 0.56 0.84 -1.09 2.23 1.00 4885 2738
muIncorrect_work_domainEmbedded 0.10 0.82 -1.48 1.67 1.00 5025 3010
muIncorrect_work_domainFinance -0.53 0.82 -2.18 1.09 1.00 4234 2849
muIncorrect_work_domainMixed 0.06 0.83 -1.55 1.74 1.00 5398 3006
muIncorrect_work_domainMusic 0.20 0.83 -1.42 1.83 1.00 5311 3096
muIncorrect_work_domainNone 0.96 0.79 -0.62 2.51 1.00 3932 2869
muIncorrect_work_domainRetail -0.14 0.85 -1.82 1.54 1.00 4728 2873
muIncorrect_work_domainTelecom -0.22 0.86 -1.89 1.46 1.00 4760 3036
muIncorrect_work_domainWeb -0.22 0.77 -1.71 1.27 1.00 5523 3074
muIncorrect_workplace_peer_reviewfalse -0.13 0.71 -1.53 1.27 1.00 4800 3081
muNone_high_debt_versionfalse 0.33 0.61 -0.86 1.53 1.00 5150 3136
muNone_work_domainDevops -0.32 0.84 -1.98 1.33 1.00 4283 2534
muNone_work_domainEmbedded -0.46 0.84 -2.07 1.21 1.00 4825 2904
muNone_work_domainFinance -0.28 0.79 -1.82 1.23 1.00 4672 3358
muNone_work_domainMixed -0.46 0.86 -2.12 1.25 1.00 4263 2263
muNone_work_domainMusic 0.06 0.84 -1.54 1.69 1.00 5427 2961
muNone_work_domainNone -0.36 0.77 -1.84 1.16 1.00 4044 2884
muNone_work_domainRetail 0.28 0.87 -1.39 1.96 1.00 4580 2720
muNone_work_domainTelecom 0.41 0.83 -1.20 2.00 1.00 4867 3193
muNone_work_domainWeb 0.02 0.72 -1.36 1.44 1.00 4756 2712
muNone_workplace_peer_reviewfalse 0.91 0.70 -0.51 2.25 1.00 3915 2584
Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).ranef(documentation2)$session
, , muIncorrect_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.5030895 1.374476 -3.9307447 1.769319
6033d90a5af2c702367b3a96 -0.4158952 1.357298 -3.9386175 1.864107
6034fc165af2c702367b3a98 -0.4093931 1.403509 -4.0887083 1.982395
603500725af2c702367b3a99 1.1780586 1.650336 -0.7426813 5.397898
603f97625af2c702367b3a9d 1.3463367 1.653553 -0.5051912 5.683918
603fd5d95af2c702367b3a9e -0.4511179 1.426560 -4.0300525 1.744010
60409b7b5af2c702367b3a9f 1.2075002 1.730038 -0.7305759 5.688347
604b82b5a7718fbed181b336 -0.4179851 1.365920 -3.9708575 1.862799
6050c1bf856f36729d2e5218 1.1310149 1.667002 -0.7899115 5.510719
6050e1e7856f36729d2e5219 0.2272325 1.047187 -1.8139178 2.718641
6055fdc6856f36729d2e521b -0.5049231 1.422157 -4.0931628 1.836192
60589862856f36729d2e521f 0.1832636 1.056718 -1.9893550 2.630138
605a30a7856f36729d2e5221 -0.6233837 1.392982 -4.3186890 1.474438
605afa3a856f36729d2e5222 -0.6292202 1.376357 -4.3781348 1.388346
605c8bc6856f36729d2e5223 0.3887177 1.121497 -1.6174260 3.033014
605f3f2d856f36729d2e5224 -0.5463267 1.331255 -3.9673140 1.487726
605f46c3856f36729d2e5225 -0.5564023 1.355038 -3.9858710 1.550545
60605337856f36729d2e5226 0.2891333 1.064092 -1.7398563 2.774680
60609ae6856f36729d2e5228 0.3382981 1.149913 -1.7230827 3.150140
6061ce91856f36729d2e522e -0.6233638 1.467581 -4.3315405 1.417251
6061f106856f36729d2e5231 -0.6354975 1.386346 -4.1724217 1.518698
6068ea9f856f36729d2e523e -0.4164789 1.401642 -3.9128615 1.872550
, , muNone_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.24695357 1.057056 -2.6432085 1.9416808
6033d90a5af2c702367b3a96 0.76402720 1.476513 -1.1382385 4.6813198
6034fc165af2c702367b3a98 0.74398084 1.447418 -1.0958607 4.6706018
603500725af2c702367b3a99 -0.71170659 1.415355 -4.4113902 1.1574823
603f97625af2c702367b3a9d -0.78746011 1.484336 -4.6803907 1.1457588
603fd5d95af2c702367b3a9e 0.76024499 1.371167 -0.9772659 4.5589980
60409b7b5af2c702367b3a9f -0.71559812 1.396389 -4.3269297 1.1831625
604b82b5a7718fbed181b336 0.72385145 1.400408 -1.1500303 4.5489147
6050c1bf856f36729d2e5218 -0.61485084 1.421435 -4.1616568 1.5695170
6050e1e7856f36729d2e5219 -0.77352130 1.445093 -4.5717207 1.1377753
6055fdc6856f36729d2e521b 0.90884672 1.505380 -0.8858834 4.9570738
60589862856f36729d2e521f -0.78531000 1.464251 -4.5601387 1.1346605
605a30a7856f36729d2e5221 -0.04501468 1.029336 -2.2631813 2.0938388
605afa3a856f36729d2e5222 0.02082146 1.006806 -2.0929242 2.3135298
605c8bc6856f36729d2e5223 0.14621432 1.083157 -2.0527870 2.6032780
605f3f2d856f36729d2e5224 0.98109241 1.482548 -0.7751370 5.0230745
605f46c3856f36729d2e5225 0.98534284 1.475205 -0.7205648 4.6993383
60605337856f36729d2e5226 -0.87094825 1.405642 -4.5450395 0.9272302
60609ae6856f36729d2e5228 0.15423182 1.021028 -1.8388072 2.6485820
6061ce91856f36729d2e522e -0.01613100 1.006420 -2.1375935 2.3118230
6061f106856f36729d2e5231 0.98003671 1.558823 -0.9379544 5.1424430
6068ea9f856f36729d2e523e 0.73797729 1.520678 -1.2018937 4.5241027plot(documentation2, ask = FALSE)
pp_check(documentation2, nsamples = 150, type= "bars")
documentation3 <- documentation0.with(c("workplace_peer_review", "work_domain", "workplace_td_tracking"))Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.summary(documentation3) Family: categorical
Links: muIncorrect = logit; muNone = logit
Formula: documentation ~ 1 + high_debt_version + (1 | session) + work_domain + workplace_peer_review + workplace_td_tracking
Data: as.data.frame(data) (Number of observations: 44)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~session (Number of levels: 22)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept) 1.20 0.93 0.04 3.44 1.00 910 1621
sd(muNone_Intercept) 1.13 0.95 0.03 3.47 1.00 661 1776
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept -0.14 0.87 -1.88 1.54 1.00 3984 3109
muNone_Intercept 0.92 0.81 -0.70 2.52 1.00 4890 3064
muIncorrect_high_debt_versionfalse -0.09 0.63 -1.32 1.11 1.00 5865 3235
muIncorrect_work_domainDevops 0.53 0.85 -1.15 2.20 1.00 5481 3312
muIncorrect_work_domainEmbedded 0.16 0.83 -1.46 1.80 1.00 6201 3156
muIncorrect_work_domainFinance -0.49 0.82 -2.13 1.08 1.00 5218 3204
muIncorrect_work_domainMixed 0.05 0.82 -1.60 1.68 1.00 5359 2995
muIncorrect_work_domainMusic 0.27 0.84 -1.38 1.92 1.00 5048 2756
muIncorrect_work_domainNone 0.90 0.77 -0.64 2.41 1.00 4973 2988
muIncorrect_work_domainRetail -0.13 0.87 -1.87 1.61 1.00 6895 3000
muIncorrect_work_domainTelecom -0.18 0.86 -1.81 1.57 1.00 6019 2789
muIncorrect_work_domainWeb -0.28 0.73 -1.78 1.13 1.00 5488 3014
muIncorrect_workplace_peer_reviewfalse -0.20 0.74 -1.67 1.26 1.00 5328 3236
muIncorrect_workplace_td_trackingfalse 0.44 0.73 -0.98 1.88 1.00 5437 3197
muNone_high_debt_versionfalse 0.30 0.61 -0.86 1.51 1.00 6084 3034
muNone_work_domainDevops -0.30 0.86 -2.00 1.42 1.00 6109 2828
muNone_work_domainEmbedded -0.53 0.88 -2.23 1.20 1.00 4530 2912
muNone_work_domainFinance -0.31 0.77 -1.81 1.20 1.00 4634 3020
muNone_work_domainMixed -0.42 0.85 -2.05 1.27 1.00 6175 2912
muNone_work_domainMusic -0.01 0.83 -1.65 1.64 1.00 5959 3222
muNone_work_domainNone -0.31 0.78 -1.82 1.20 1.00 5372 2771
muNone_work_domainRetail 0.23 0.86 -1.45 1.91 1.00 6984 2703
muNone_work_domainTelecom 0.36 0.83 -1.31 1.98 1.00 6030 2925
muNone_work_domainWeb 0.06 0.72 -1.35 1.51 1.00 5674 2852
muNone_workplace_peer_reviewfalse 0.99 0.73 -0.50 2.35 1.00 3836 2466
muNone_workplace_td_trackingfalse -0.60 0.71 -2.02 0.79 1.00 5115 3024
Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).ranef(documentation3)$session
, , muIncorrect_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.5603771 1.365593 -4.1833287 1.536481
6033d90a5af2c702367b3a96 -0.4378709 1.411105 -4.0232000 1.953867
6034fc165af2c702367b3a98 -0.4246564 1.325774 -3.9571182 1.797831
603500725af2c702367b3a99 1.1527434 1.630248 -0.7546270 5.406999
603f97625af2c702367b3a9d 1.2640296 1.603604 -0.6170976 5.220101
603fd5d95af2c702367b3a9e -0.4376334 1.360366 -3.9066025 1.801434
60409b7b5af2c702367b3a9f 1.1396438 1.583356 -0.7162260 5.219274
604b82b5a7718fbed181b336 -0.2858046 1.401094 -3.7536563 2.331719
6050c1bf856f36729d2e5218 1.0381249 1.550079 -0.8516789 5.075871
6050e1e7856f36729d2e5219 0.1419191 1.043177 -1.9406677 2.579764
6055fdc6856f36729d2e521b -0.3934914 1.372011 -3.8387867 2.017171
60589862856f36729d2e521f 0.3169621 1.113475 -1.7286922 2.969353
605a30a7856f36729d2e5221 -0.6565622 1.361179 -4.3442972 1.346198
605afa3a856f36729d2e5222 -0.5458650 1.363074 -4.0513580 1.581949
605c8bc6856f36729d2e5223 0.5259550 1.185150 -1.4160845 3.495365
605f3f2d856f36729d2e5224 -0.4859145 1.403190 -4.2464295 1.714097
605f46c3856f36729d2e5225 -0.6173888 1.413211 -4.3276640 1.477781
60605337856f36729d2e5226 0.2000773 1.007543 -1.7251890 2.564823
60609ae6856f36729d2e5228 0.3407595 1.099734 -1.5694170 2.974571
6061ce91856f36729d2e522e -0.6479877 1.406089 -4.3282187 1.342581
6061f106856f36729d2e5231 -0.6017436 1.359694 -4.0783642 1.402426
6068ea9f856f36729d2e523e -0.2822018 1.393687 -3.6743192 2.224814
, , muNone_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.16947641 1.0320587 -2.5085847 1.920971
6033d90a5af2c702367b3a96 0.76713004 1.4676916 -1.1174510 4.884850
6034fc165af2c702367b3a98 0.73337699 1.3911643 -1.0973840 4.749686
603500725af2c702367b3a99 -0.66424554 1.3994763 -4.3233720 1.360758
603f97625af2c702367b3a9d -0.70561958 1.4018144 -4.4437248 1.253827
603fd5d95af2c702367b3a9e 0.77288448 1.4467877 -0.9924838 4.579692
60409b7b5af2c702367b3a9f -0.66309446 1.4183946 -4.2448305 1.230885
604b82b5a7718fbed181b336 0.59804841 1.4144994 -1.3090165 4.335915
6050c1bf856f36729d2e5218 -0.52549501 1.3705740 -4.0863820 1.557628
6050e1e7856f36729d2e5219 -0.70485253 1.4192676 -4.5767460 1.139698
6055fdc6856f36729d2e521b 0.75187230 1.4520325 -1.1047047 4.681177
60589862856f36729d2e521f -0.86499771 1.4999719 -4.7709317 1.007139
605a30a7856f36729d2e5221 0.06077508 0.9843348 -2.0622470 2.409156
605afa3a856f36729d2e5222 -0.10507117 1.0078779 -2.4009560 1.917847
605c8bc6856f36729d2e5223 0.07513788 1.0418569 -2.0206677 2.581048
605f3f2d856f36729d2e5224 0.80558375 1.3966626 -0.8747045 4.511000
605f46c3856f36729d2e5225 1.02366435 1.4866180 -0.7480916 4.930530
60605337856f36729d2e5226 -0.74820986 1.3894169 -4.4882753 1.038718
60609ae6856f36729d2e5228 0.20695472 1.0654662 -1.8987093 2.793441
6061ce91856f36729d2e522e 0.04172705 0.9641196 -1.9494337 2.214633
6061f106856f36729d2e5231 0.97374392 1.5107063 -0.7127553 5.192776
6068ea9f856f36729d2e523e 0.59649728 1.4426868 -1.4235145 4.412081plot(documentation3, ask = FALSE)
pp_check(documentation3, nsamples = 150, type= "bars")
Final model
All models have sampled okey but documentation3 has a bitter fit for the high debt beta parameter is significantly better than some of the simple models according to loo. wee choose documentation3 as out final model.
Trying to add covariance
documentation3_with_c <- brm(
"documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | c | session)",
prior = c(
prior(normal(0, 0.9), class = "b"),
prior(normal(0, 0.9), class = "Intercept"),
prior(exponential(1), class = "sd", dpar = "muIncorrect"),
prior(exponential(1), class = "sd", dpar = "muNone"),
prior(lkj(2), class = "L",)
),
family = categorical(),
data = as.data.frame(d.both_completed),
file = "fits/documentation3_with_c",
file_refit = "on_change"
)Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Compiling Stan program...
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
\
|
/
-
Start samplingRunning MCMC with 4 parallel chains...
Chain 1 Iteration: 1 / 2000 [ 0%] (Warmup)
Chain 1 Iteration: 100 / 2000 [ 5%] (Warmup)
Chain 1 Iteration: 200 / 2000 [ 10%] (Warmup)
Chain 2 Iteration: 1 / 2000 [ 0%] (Warmup)
Chain 2 Iteration: 100 / 2000 [ 5%] (Warmup)
Chain 2 Iteration: 200 / 2000 [ 10%] (Warmup)
Chain 3 Iteration: 1 / 2000 [ 0%] (Warmup)
Chain 3 Iteration: 100 / 2000 [ 5%] (Warmup)
Chain 3 Iteration: 200 / 2000 [ 10%] (Warmup)
Chain 4 Iteration: 1 / 2000 [ 0%] (Warmup)
Chain 4 Iteration: 100 / 2000 [ 5%] (Warmup)
Chain 4 Iteration: 200 / 2000 [ 10%] (Warmup)
Chain 4 Iteration: 300 / 2000 [ 15%] (Warmup)
Chain 1 Iteration: 300 / 2000 [ 15%] (Warmup)
Chain 1 Iteration: 400 / 2000 [ 20%] (Warmup)
Chain 1 Iteration: 500 / 2000 [ 25%] (Warmup)
Chain 1 Iteration: 600 / 2000 [ 30%] (Warmup)
Chain 1 Iteration: 700 / 2000 [ 35%] (Warmup)
Chain 2 Iteration: 300 / 2000 [ 15%] (Warmup)
Chain 2 Iteration: 400 / 2000 [ 20%] (Warmup)
Chain 2 Iteration: 500 / 2000 [ 25%] (Warmup)
Chain 3 Iteration: 300 / 2000 [ 15%] (Warmup)
Chain 3 Iteration: 400 / 2000 [ 20%] (Warmup)
Chain 4 Iteration: 400 / 2000 [ 20%] (Warmup)
Chain 4 Iteration: 500 / 2000 [ 25%] (Warmup)
Chain 1 Iteration: 800 / 2000 [ 40%] (Warmup)
Chain 2 Iteration: 600 / 2000 [ 30%] (Warmup)
Chain 3 Iteration: 500 / 2000 [ 25%] (Warmup)
Chain 4 Iteration: 600 / 2000 [ 30%] (Warmup)
Chain 1 Iteration: 900 / 2000 [ 45%] (Warmup)
Chain 1 Iteration: 1000 / 2000 [ 50%] (Warmup)
Chain 2 Iteration: 700 / 2000 [ 35%] (Warmup)
Chain 2 Iteration: 800 / 2000 [ 40%] (Warmup)
Chain 3 Iteration: 600 / 2000 [ 30%] (Warmup)
Chain 4 Iteration: 700 / 2000 [ 35%] (Warmup)
Chain 4 Iteration: 800 / 2000 [ 40%] (Warmup)
Chain 1 Iteration: 1001 / 2000 [ 50%] (Sampling)
Chain 2 Iteration: 900 / 2000 [ 45%] (Warmup)
Chain 3 Iteration: 700 / 2000 [ 35%] (Warmup)
Chain 4 Iteration: 900 / 2000 [ 45%] (Warmup)
Chain 1 Iteration: 1100 / 2000 [ 55%] (Sampling)
Chain 2 Iteration: 1000 / 2000 [ 50%] (Warmup)
Chain 2 Iteration: 1001 / 2000 [ 50%] (Sampling)
Chain 3 Iteration: 800 / 2000 [ 40%] (Warmup)
Chain 3 Iteration: 900 / 2000 [ 45%] (Warmup)
Chain 4 Iteration: 1000 / 2000 [ 50%] (Warmup)
Chain 4 Iteration: 1001 / 2000 [ 50%] (Sampling)
Chain 1 Iteration: 1200 / 2000 [ 60%] (Sampling)
Chain 3 Iteration: 1000 / 2000 [ 50%] (Warmup)
Chain 3 Iteration: 1001 / 2000 [ 50%] (Sampling)
Chain 4 Iteration: 1100 / 2000 [ 55%] (Sampling)
Chain 1 Iteration: 1300 / 2000 [ 65%] (Sampling)
Chain 2 Iteration: 1100 / 2000 [ 55%] (Sampling)
Chain 3 Iteration: 1100 / 2000 [ 55%] (Sampling)
Chain 4 Iteration: 1200 / 2000 [ 60%] (Sampling)
Chain 4 Iteration: 1300 / 2000 [ 65%] (Sampling)
Chain 1 Iteration: 1400 / 2000 [ 70%] (Sampling)
Chain 2 Iteration: 1200 / 2000 [ 60%] (Sampling)
Chain 3 Iteration: 1200 / 2000 [ 60%] (Sampling)
Chain 1 Iteration: 1500 / 2000 [ 75%] (Sampling)
Chain 2 Iteration: 1300 / 2000 [ 65%] (Sampling)
Chain 3 Iteration: 1300 / 2000 [ 65%] (Sampling)
Chain 4 Iteration: 1400 / 2000 [ 70%] (Sampling)
Chain 1 Iteration: 1600 / 2000 [ 80%] (Sampling)
Chain 3 Iteration: 1400 / 2000 [ 70%] (Sampling)
Chain 4 Iteration: 1500 / 2000 [ 75%] (Sampling)
Chain 1 Iteration: 1700 / 2000 [ 85%] (Sampling)
Chain 2 Iteration: 1400 / 2000 [ 70%] (Sampling)
Chain 3 Iteration: 1500 / 2000 [ 75%] (Sampling)
Chain 4 Iteration: 1600 / 2000 [ 80%] (Sampling)
Chain 1 Iteration: 1800 / 2000 [ 90%] (Sampling)
Chain 2 Iteration: 1500 / 2000 [ 75%] (Sampling)
Chain 3 Iteration: 1600 / 2000 [ 80%] (Sampling)
Chain 4 Iteration: 1700 / 2000 [ 85%] (Sampling)
Chain 1 Iteration: 1900 / 2000 [ 95%] (Sampling)
Chain 3 Iteration: 1700 / 2000 [ 85%] (Sampling)
Chain 4 Iteration: 1800 / 2000 [ 90%] (Sampling)
Chain 2 Iteration: 1600 / 2000 [ 80%] (Sampling)
Chain 4 Iteration: 1900 / 2000 [ 95%] (Sampling)
Chain 1 Iteration: 2000 / 2000 [100%] (Sampling)
Chain 3 Iteration: 1800 / 2000 [ 90%] (Sampling)
Chain 1 finished in 2.2 seconds.
Chain 2 Iteration: 1700 / 2000 [ 85%] (Sampling)
Chain 3 Iteration: 1900 / 2000 [ 95%] (Sampling)
Chain 4 Iteration: 2000 / 2000 [100%] (Sampling)
Chain 4 finished in 2.2 seconds.
Chain 2 Iteration: 1800 / 2000 [ 90%] (Sampling)
Chain 3 Iteration: 2000 / 2000 [100%] (Sampling)
Chain 3 finished in 2.4 seconds.
Chain 2 Iteration: 1900 / 2000 [ 95%] (Sampling)
Chain 2 Iteration: 2000 / 2000 [100%] (Sampling)
Chain 2 finished in 2.7 seconds.
All 4 chains finished successfully.
Mean chain execution time: 2.4 seconds.
Total execution time: 2.9 seconds.Registered S3 method overwritten by 'data.table':
method from
print.data.table summary(documentation3_with_c) Family: categorical
Links: muIncorrect = logit; muNone = logit
Formula: documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | c | session)
Data: as.data.frame(d.both_completed) (Number of observations: 44)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~session (Number of levels: 22)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept) 1.22 0.97 0.04 3.59 1.00 1141 1987
sd(muNone_Intercept) 1.21 0.95 0.04 3.60 1.00 1238 2277
cor(muIncorrect_Intercept,muNone_Intercept) -0.17 0.43 -0.86 0.72 1.00 2419 2619
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept -0.18 0.89 -1.95 1.54 1.00 5323 3288
muNone_Intercept 0.85 0.80 -0.76 2.38 1.00 5920 3224
muIncorrect_high_debt_versionfalse -0.09 0.63 -1.34 1.13 1.00 8422 2880
muIncorrect_workplace_peer_reviewfalse -0.15 0.75 -1.64 1.34 1.00 6767 2502
muIncorrect_work_domainDevops 0.51 0.86 -1.16 2.20 1.00 6404 2953
muIncorrect_work_domainEmbedded 0.14 0.83 -1.51 1.76 1.00 6522 2842
muIncorrect_work_domainFinance -0.51 0.83 -2.15 1.14 1.00 7026 2872
muIncorrect_work_domainMixed 0.04 0.83 -1.63 1.66 1.00 8060 2902
muIncorrect_work_domainMusic 0.26 0.86 -1.45 2.00 1.00 7085 2696
muIncorrect_work_domainNone 0.89 0.78 -0.65 2.37 1.00 5873 3152
muIncorrect_work_domainRetail -0.12 0.90 -1.84 1.62 1.00 7808 3043
muIncorrect_work_domainTelecom -0.17 0.86 -1.83 1.51 1.00 7942 3004
muIncorrect_work_domainWeb -0.27 0.74 -1.78 1.21 1.00 6352 3191
muIncorrect_workplace_td_trackingfalse 0.44 0.76 -1.07 1.93 1.00 6027 2948
muNone_high_debt_versionfalse 0.32 0.60 -0.84 1.51 1.00 6829 2873
muNone_workplace_peer_reviewfalse 1.00 0.72 -0.39 2.35 1.00 5148 3092
muNone_work_domainDevops -0.27 0.86 -1.93 1.43 1.00 6542 2769
muNone_work_domainEmbedded -0.50 0.84 -2.15 1.16 1.00 6381 3161
muNone_work_domainFinance -0.30 0.79 -1.84 1.30 1.00 6143 3353
muNone_work_domainMixed -0.41 0.85 -2.08 1.25 1.00 6550 2801
muNone_work_domainMusic -0.01 0.84 -1.63 1.59 1.00 7421 3059
muNone_work_domainNone -0.27 0.79 -1.78 1.25 1.00 6824 3123
muNone_work_domainRetail 0.22 0.87 -1.46 1.93 1.00 7755 2985
muNone_work_domainTelecom 0.33 0.86 -1.32 2.06 1.00 6296 2728
muNone_work_domainWeb 0.05 0.73 -1.38 1.48 1.00 7448 2919
muNone_workplace_td_trackingfalse -0.57 0.71 -1.99 0.84 1.00 6388 3297
Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).ranef(documentation3_with_c)$session
, , muIncorrect_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.4740170 1.349549 -3.8904217 1.622149
6033d90a5af2c702367b3a96 -0.5486771 1.422207 -4.2951387 1.788122
6034fc165af2c702367b3a98 -0.5364618 1.420365 -4.3462845 1.694847
603500725af2c702367b3a99 1.1888933 1.665348 -0.6752396 5.454351
603f97625af2c702367b3a9d 1.2871637 1.554135 -0.5700177 5.168141
603fd5d95af2c702367b3a9e -0.5383334 1.395016 -4.1240832 1.656860
60409b7b5af2c702367b3a9f 1.1492655 1.557404 -0.6702289 5.124011
604b82b5a7718fbed181b336 -0.3947439 1.428949 -3.9845432 2.057823
6050c1bf856f36729d2e5218 1.0869348 1.574547 -0.6869054 5.109583
6050e1e7856f36729d2e5219 0.2284617 1.050561 -1.8104610 2.668000
6055fdc6856f36729d2e521b -0.5149120 1.391413 -4.0554980 1.654794
60589862856f36729d2e521f 0.4047294 1.119904 -1.6248867 3.103528
605a30a7856f36729d2e5221 -0.6305685 1.385323 -4.2688537 1.414821
605afa3a856f36729d2e5222 -0.5260204 1.350655 -4.0110530 1.556350
605c8bc6856f36729d2e5223 0.4871209 1.183538 -1.4799500 3.482942
605f3f2d856f36729d2e5224 -0.5572856 1.383517 -4.0131750 1.559544
605f46c3856f36729d2e5225 -0.7259060 1.464672 -4.3730900 1.305701
60605337856f36729d2e5226 0.3123440 1.023106 -1.5360945 2.745020
60609ae6856f36729d2e5228 0.2911768 1.105874 -1.7371525 2.962782
6061ce91856f36729d2e522e -0.6295638 1.354371 -4.1986288 1.284592
6061f106856f36729d2e5231 -0.7161355 1.403589 -4.4195718 1.209440
6068ea9f856f36729d2e523e -0.4117994 1.434870 -4.0406335 2.093152
, , muNone_Intercept
Estimate Est.Error Q2.5 Q97.5
6033d69a5af2c702367b3a95 -0.095175704 1.0282401 -2.4283315 2.0582043
6033d90a5af2c702367b3a96 0.869406506 1.4682907 -1.0033360 4.5763228
6034fc165af2c702367b3a98 0.826466220 1.5007878 -1.1077500 4.6382268
603500725af2c702367b3a99 -0.830604253 1.4675248 -4.6771415 1.1719215
603f97625af2c702367b3a9d -0.866577467 1.4771711 -4.8287575 1.1342350
603fd5d95af2c702367b3a9e 0.864852450 1.5246855 -1.0572250 4.8986993
60409b7b5af2c702367b3a9f -0.831289932 1.4408650 -4.5627458 1.0589548
604b82b5a7718fbed181b336 0.671833314 1.4217659 -1.3227875 4.3870320
6050c1bf856f36729d2e5218 -0.687363331 1.4440616 -4.3477467 1.4542002
6050e1e7856f36729d2e5219 -0.717601888 1.4426154 -4.5868553 1.2550483
6055fdc6856f36729d2e521b 0.876459045 1.4810539 -1.0061095 4.7831158
60589862856f36729d2e521f -0.860613797 1.4319841 -4.3080075 1.0334343
605a30a7856f36729d2e5221 0.107802137 0.9669728 -1.8747055 2.2739908
605afa3a856f36729d2e5222 -0.023811007 1.0210658 -2.2640835 2.2188023
605c8bc6856f36729d2e5223 0.005140308 1.0488520 -2.2531653 2.3370798
605f3f2d856f36729d2e5224 0.902602636 1.4443933 -0.8216154 4.7493038
605f46c3856f36729d2e5225 1.086796364 1.4870779 -0.6899161 4.8028003
60605337856f36729d2e5226 -0.795304877 1.3837966 -4.4117265 0.9448756
60609ae6856f36729d2e5228 0.129812004 1.0106389 -1.8727820 2.4984315
6061ce91856f36729d2e522e 0.109028807 0.9949454 -1.9458302 2.3347675
6061f106856f36729d2e5231 1.038764660 1.5270026 -0.7500039 4.9471303
6068ea9f856f36729d2e523e 0.700633994 1.4825041 -1.3700782 4.6034325plot(documentation3_with_c, ask = FALSE)
pp_check(documentation3_with_c, nsamples = 150, type= "bars")
loo(
documentation3,
documentation3_with_c
)Found 4 observations with a pareto_k > 0.7 in model 'documentation3'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Found 3 observations with a pareto_k > 0.7 in model 'documentation3_with_c'. It is recommended to set 'moment_match = TRUE' in order to perform moment matching for problematic observations. Output of model 'documentation3':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 29 65.9% 1023
(0.5, 0.7] (ok) 11 25.0% 675
(0.7, 1] (bad) 4 9.1% 245
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Output of model 'documentation3_with_c':
Computed from 4000 by 44 log-likelihood matrix------
Monte Carlo SE of elpd_loo is NA.
Pareto k diagnostic values:
Count Pct. Min. n_eff
(-Inf, 0.5] (good) 28 63.6% 835
(0.5, 0.7] (ok) 13 29.5% 458
(0.7, 1] (bad) 3 6.8% 367
(1, Inf) (very bad) 0 0.0% <NA>
See help('pareto-k-diagnostic') for details.
Model comparisons:
elpd_diff se_diff
documentation3_with_c 0.0 0.0
documentation3 -0.3 0.3 Adding the covariance parameter did not significantly improve the model and we will therefore continue to use the simpler model documentation3.
Model with with incomplete data points
Some participants did only complete one scenario. Those has been excluded from the initial dataset to improve sampling of the models. We do however want to use all data we can and will therefore try to fit the model with the complete dataset.
documentation3.all <- brm(
"documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | session)",
prior = c(
prior(normal(0, 0.9), class = "b"),
prior(normal(0, 0.9), class = "Intercept"),
prior(exponential(1), class = "sd", dpar = "muIncorrect"),
prior(exponential(1), class = "sd", dpar = "muNone")
),
family = categorical(),
data = as.data.frame(d.completed),
file = "fits/documentation3.all",
file_refit = "on_change"
)Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.Specifying global priors for regression coefficients in categorical models is deprecated. Please specify priors separately for each response category.summary(documentation3.all) Family: categorical
Links: muIncorrect = logit; muNone = logit
Formula: documentation ~ 1 + high_debt_version + workplace_peer_review + work_domain + workplace_td_tracking + (1 | session)
Data: as.data.frame(d.completed) (Number of observations: 49)
Samples: 4 chains, each with iter = 2000; warmup = 1000; thin = 1;
total post-warmup samples = 4000
Group-Level Effects:
~session (Number of levels: 27)
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
sd(muIncorrect_Intercept) 1.06 0.88 0.03 3.24 1.00 998 1808
sd(muNone_Intercept) 0.99 0.86 0.03 3.25 1.01 855 1697
Population-Level Effects:
Estimate Est.Error l-95% CI u-95% CI Rhat Bulk_ESS Tail_ESS
muIncorrect_Intercept 0.00 0.88 -1.78 1.65 1.00 3786 2586
muNone_Intercept 0.77 0.81 -0.82 2.33 1.00 3981 2632
muIncorrect_high_debt_versionfalse -0.35 0.64 -1.63 0.91 1.00 3879 2558
muIncorrect_workplace_peer_reviewfalse -0.06 0.71 -1.45 1.33 1.00 4096 2720
muIncorrect_work_domainAutomotive -0.49 0.78 -2.03 1.02 1.00 4267 2655
muIncorrect_work_domainDevops 0.51 0.85 -1.16 2.15 1.00 5133 2988
muIncorrect_work_domainEMCommerce 0.30 0.88 -1.43 2.04 1.00 5377 2845
muIncorrect_work_domainEmbedded 0.17 0.85 -1.45 1.83 1.00 5257 2458
muIncorrect_work_domainFinance -0.54 0.82 -2.15 1.09 1.00 3847 2714
muIncorrect_work_domainMixed 0.02 0.84 -1.62 1.63 1.00 5104 2788
muIncorrect_work_domainMusic 0.28 0.84 -1.40 1.91 1.00 5266 2831
muIncorrect_work_domainNone 0.86 0.73 -0.56 2.29 1.00 3809 2573
muIncorrect_work_domainRetail -0.16 0.86 -1.88 1.52 1.00 5015 2441
muIncorrect_work_domainTelecom -0.22 0.85 -1.88 1.44 1.00 5254 2855
muIncorrect_work_domainWeb -0.44 0.73 -1.86 0.95 1.00 5044 3030
muIncorrect_workplace_td_trackingfalse 0.58 0.75 -0.89 2.01 1.00 4111 2895
muNone_high_debt_versionfalse 0.37 0.60 -0.81 1.57 1.00 3991 3132
muNone_workplace_peer_reviewfalse 1.00 0.69 -0.39 2.31 1.00 3357 2924
muNone_work_domainAutomotive 0.76 0.72 -0.69 2.13 1.00 4474 2700
muNone_work_domainDevops -0.27 0.86 -1.97 1.38 1.00 4875 2761
muNone_work_domainEMCommerce -0.15 0.86 -1.87 1.54 1.00 5608 3018
muNone_work_domainEmbedded -0.53 0.86 -2.22 1.13 1.00 4529 3115
muNone_work_domainFinance -0.24 0.79 -1.79 1.30 1.00 4305 2587
muNone_work_domainMixed -0.35 0.87 -2.10 1.33 1.00 4934 3109
muNone_work_domainMusic -0.00 0.86 -1.70 1.72 1.00 4805 2833
muNone_work_domainNone -0.05 0.73 -1.43 1.38 1.00 4750 2996
muNone_work_domainRetail 0.25 0.85 -1.37 1.87 1.00 5266 2843
muNone_work_domainTelecom 0.40 0.87 -1.27 2.11 1.00 5359 3074
muNone_work_domainWeb 0.40 0.69 -0.99 1.79 1.00 4645 3143
muNone_workplace_td_trackingfalse -0.84 0.71 -2.24 0.57 1.00 3667 2220
Samples were drawn using sample(hmc). For each parameter, Bulk_ESS
and Tail_ESS are effective sample size measures, and Rhat is the potential
scale reduction factor on split chains (at convergence, Rhat = 1).ranef(documentation3.all)$session
, , muIncorrect_Intercept
Estimate Est.Error Q2.5 Q97.5
6033c6fc5af2c702367b3a93 0.56668439 1.3900710 -1.3564902 4.358797
6033d69a5af2c702367b3a95 -0.51816620 1.2588619 -3.9174198 1.362781
6033d90a5af2c702367b3a96 -0.30818290 1.2365050 -3.3053398 1.836268
6034fc165af2c702367b3a98 -0.35746056 1.2515824 -3.5075447 1.687832
603500725af2c702367b3a99 0.91078960 1.4807329 -0.8130972 5.051990
603f84f15af2c702367b3a9b -0.46378477 1.3007111 -3.8238760 1.598761
603f97625af2c702367b3a9d 1.09826959 1.4915761 -0.5676147 5.028738
603fd5d95af2c702367b3a9e -0.32641539 1.2325152 -3.4746550 1.755247
60409b7b5af2c702367b3a9f 0.90267517 1.4050692 -0.7899109 4.615886
604b82b5a7718fbed181b336 -0.21797633 1.2297951 -3.2245887 1.984291
604f1239a7718fbed181b33f 0.57575704 1.3356545 -1.2706010 4.265109
6050c1bf856f36729d2e5218 0.84327015 1.4497551 -0.9712524 4.842020
6050e1e7856f36729d2e5219 0.09232487 1.0199872 -1.8919607 2.392772
6055fdc6856f36729d2e521b -0.35640012 1.2716045 -3.5745012 1.728544
60579f2a856f36729d2e521e -0.29381870 1.2108395 -3.2560522 1.912461
60589862856f36729d2e521f 0.26026107 1.0077301 -1.6189578 2.779680
605a30a7856f36729d2e5221 -0.58023210 1.2833446 -4.0179022 1.251975
605afa3a856f36729d2e5222 -0.47011977 1.3125490 -3.9474637 1.515763
605c8bc6856f36729d2e5223 0.42659282 1.0888754 -1.2925652 3.231538
605f3f2d856f36729d2e5224 -0.29479752 1.2642510 -3.3185883 1.942462
605f46c3856f36729d2e5225 -0.44612047 1.2838081 -3.8377990 1.523013
60605337856f36729d2e5226 0.34015025 1.0587691 -1.5499285 3.050282
60609ae6856f36729d2e5228 0.16906903 0.9865542 -1.7369890 2.639110
6061ce91856f36729d2e522e -0.56594985 1.2566168 -3.9529910 1.183943
6061f106856f36729d2e5231 -0.62303299 1.2602445 -3.9279680 1.081355
60672faa856f36729d2e523c -0.34915974 1.2340610 -3.4919848 1.700463
6068ea9f856f36729d2e523e -0.29085361 1.3114368 -3.6383952 1.900159
, , muNone_Intercept
Estimate Est.Error Q2.5 Q97.5
6033c6fc5af2c702367b3a93 -0.31975767 1.2189100 -3.3697577 1.6906123
6033d69a5af2c702367b3a95 -0.06278226 0.9273268 -2.1735618 1.9110213
6033d90a5af2c702367b3a96 0.58113279 1.3059730 -1.0898352 4.0925765
6034fc165af2c702367b3a98 0.62740112 1.2820138 -1.0774657 4.0800975
603500725af2c702367b3a99 -0.50791441 1.1806103 -3.6122665 1.1815005
603f84f15af2c702367b3a9b -0.42847632 1.2603699 -3.8646552 1.4205428
603f97625af2c702367b3a9d -0.59629911 1.2798042 -4.1252695 1.0923448
603fd5d95af2c702367b3a9e 0.55655334 1.2269826 -1.1445050 3.9102400
60409b7b5af2c702367b3a9f -0.52608135 1.2255266 -3.8040947 1.2553955
604b82b5a7718fbed181b336 0.47987881 1.2396653 -1.2894017 3.7864998
604f1239a7718fbed181b33f -0.29773930 1.2350288 -3.4176547 1.8673838
6050c1bf856f36729d2e5218 -0.41232613 1.2413335 -3.7342760 1.4039380
6050e1e7856f36729d2e5219 -0.53521184 1.2406922 -3.6471362 1.2327300
6055fdc6856f36729d2e521b 0.63073666 1.3052796 -1.1016473 4.1732833
60579f2a856f36729d2e521e 0.50515355 1.2151732 -1.2383735 3.7816730
60589862856f36729d2e521f -0.70828053 1.3667908 -4.4140030 0.9217694
605a30a7856f36729d2e5221 0.02511206 0.9109578 -1.9496705 2.0509803
605afa3a856f36729d2e5222 -0.06936711 0.9714361 -2.2305623 1.9991320
605c8bc6856f36729d2e5223 0.06387040 0.9649576 -1.8657480 2.3023402
605f3f2d856f36729d2e5224 0.52634041 1.2488189 -1.2207750 3.7981818
605f46c3856f36729d2e5225 0.72517113 1.2475205 -0.8546529 4.0628638
60605337856f36729d2e5226 -0.74071093 1.2736229 -4.1198245 0.7786216
60609ae6856f36729d2e5228 0.20337996 0.9805332 -1.6567338 2.5486325
6061ce91856f36729d2e522e 0.05958744 0.8940686 -1.8627612 2.0501413
6061f106856f36729d2e5231 0.86014412 1.4043814 -0.7155665 4.7656840
60672faa856f36729d2e523c 0.53896524 1.2493469 -1.2020765 4.0047203
6068ea9f856f36729d2e523e 0.48951876 1.2275134 -1.3053530 3.7424770plot(documentation3.all, ask = FALSE)
pp_check(documentation3_with_c, nsamples = 150, type= "bars")
Training the model on all data points reduces the uncertainty and did not result in any sampling problems. We will proceed with the model fitted to all the data.
Interpreting the model
Extract posterior samples:
post <- posterior_predict(documentation3.all, newdata = data.frame(
high_debt_version = c("false", "true"),
session = NA,
workplace_peer_review = NA,
work_domain = NA,
workplace_td_tracking = NA
))
post.low <- post[,1]
post.high <- post[,2]
summary(post) V1 V2
Min. :1.000 Min. :1.000
1st Qu.:2.000 1st Qu.:2.000
Median :3.000 Median :2.000
Mean :2.383 Mean :2.268
3rd Qu.:3.000 3rd Qu.:3.000
Max. :3.000 Max. :3.000 LS0tCnRpdGxlOiAiRG9jdW1lbnRhdGlvbiIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKIyMgTG9va2luZyBhdCB0aGUgZGF0YQpXZSBwbG90IHRoZSBkYXRhIGFuZCBjYW4gc2VlIHRoYXQgdGhlcmUgaXMgbm8gb2J2aW91cyBsYXJnZSBkaWZmZXJlbmNlIGJldHdlZW4gdGhlIGRlYnQgdmVyc2lvbnMuCgpgYGB7cn0KZC5ib3RoX2NvbXBsZXRlZCAlPiUKICBnZ3Bsb3QoYWVzKGhpZ2hfZGVidF92ZXJzaW9uLCBmaWxsID0gZG9jdW1lbnRhdGlvbikpICsKICBnZW9tX2Jhcihwb3NpdGlvbiA9ICJmaWxsIikgKwogIHNjYWxlX3lfcmV2ZXJzZSgpICsKICBzY2FsZV9maWxsX21hbnVhbCgibGVnZW5kIiwgdmFsdWVzID0gYygiQ29ycmVjdCIgPSAiZ3JlZW4iLCAiTm9uZSIgPSAieWVsbG93IiwgIkluY29ycmVjdCIgPSAicmVkIiksIGd1aWRlID0gZ3VpZGVfbGVnZW5kKHJldmVyc2UgPSBUUlVFKSkKYGBgCiMjIEluaXRpYWwgbW9kZWwKVGhlIHR5cGUgb2YgdGhlIG91dGNvbWUgaXMgYWRqYWNlbnQgY2F0ZWdvcmllcyBhbmQgaXMgbW9kZWxlZCBhcyBhIGN1bXVsYXRpdmUgZmFtaWx5LiBUT0RPIEZJWCEKCldlIGluY2x1ZGUgYGhpZ2hfZGVidF92ZXJpc29uYCBhcyB3ZWxsIGFzIGEgdmFyeWluZyBpbnRlcmNlcHQgZm9yIGVhY2ggaW5kaXZpZHVhbCBpbiBvdXIgaW5pdGlhbCBtb2RlbC4KCiMjIyBTZWxlY3RpbmcgUHJpb3JzCldlIGl0ZXJhdGUgb3ZlciB0aGUgbW9kZWwgdW50aWwgd2UgaGF2ZSBzYW5lIHByaW9ycwoKYGBge3J9Cgpkb2N1bWVudGF0aW9uMC53aXRoIDwtIGV4dGVuZGFibGVfbW9kZWwoCiAgYmFzZV9uYW1lID0gImRvY3VtZW50YXRpb24wIiwKICBiYXNlX2Zvcm11bGEgPSAiZG9jdW1lbnRhdGlvbiB+IDEgKyBoaWdoX2RlYnRfdmVyc2lvbiArICgxIHwgc2Vzc2lvbikiLAogIGJhc2VfcHJpb3JzID0gYygKICAgIHByaW9yKG5vcm1hbCgwLCAwLjkpLCBjbGFzcyA9ICJiIiksCiAgICBwcmlvcihub3JtYWwoMCwgMC45KSwgY2xhc3MgPSAiSW50ZXJjZXB0IiksCiAgICBwcmlvcihleHBvbmVudGlhbCgxKSwgY2xhc3MgPSAic2QiLCBkcGFyID0gIm11SW5jb3JyZWN0IiksCiAgICBwcmlvcihleHBvbmVudGlhbCgxKSwgY2xhc3MgPSAic2QiLCBkcGFyID0gIm11Tm9uZSIpCiAgKSwKICBmYW1pbHkgPSBjYXRlZ29yaWNhbCgpLAogIGRhdGEgPSBkLmJvdGhfY29tcGxldGVkLAopCgojIERlZmF1bHQgcHJpb3JzOgpwcmlvcl9zdW1tYXJ5KGRvY3VtZW50YXRpb24wLndpdGgob25seV9wcmlvcnM9IFRSVUUpKQoKIyBPdXIgcHJpb3JzOgpwcmlvcl9zdW1tYXJ5KGRvY3VtZW50YXRpb24wLndpdGgoc2FtcGxlX3ByaW9yID0gIm9ubHkiKSkKCiMgUHJpb3IgcHJlZGljdGl2ZSBjaGVjawpwcF9jaGVjayhkb2N1bWVudGF0aW9uMC53aXRoKHNhbXBsZV9wcmlvciA9ICJvbmx5IiksIG5zYW1wbGVzID0gMjAwLCB0eXBlID0gImJhcnMiKQpgYGAKCgojIyMgTW9kZWwgZml0CldlIGNoZWNrIHRoZSBwb3N0ZXJpb3IgZGlzdHJpYnV0aW9uIGFuZCBjYW4gc2VlIHRoYXQgdGhlIG1vZGVsIHNlZW1zIHRvIGhhdmUgYmVlbiBhYmxlIHRvIGZpdCB0aGUgZGF0YSB3ZWxsCmBgYHtyfQojIFBvc3RlcmlvciBwcmVkaWN0aXZlIGNoZWNrCnBwX2NoZWNrKGRvY3VtZW50YXRpb24wLndpdGgoKSwgbnNhbXBsZXMgPSAxMDAsIHR5cGUgPSAiYmFycyIpCgpzdW1tYXJ5KGRvY3VtZW50YXRpb24wLndpdGgoKSkKYGBgCiMjIE1vZGVsIGV4dGVuc3Rpb25zCgpXZSB1c2UgbG9vIHRvIGNoZWNrIHNvbWUgcG9zc2libGUgZXh0ZW5zaW9ucyBvbiB0aGUgbW9kZWwuCgpgYGB7cn0KCmVkbHZsX3ByaW9yIDwtIGMoCiAgcHJpb3IoZGlyaWNobGV0KDIpLCBjbGFzcyA9ICJzaW1vIiwgY29lZiA9ICJtb2VkdWNhdGlvbl9sZXZlbDEiLCBkcGFyID0gIm11SW5jb3JyZWN0IiksCiAgcHJpb3IoZGlyaWNobGV0KDIpLCBjbGFzcyA9ICJzaW1vIiwgY29lZiA9ICJtb2VkdWNhdGlvbl9sZXZlbDEiLCBkcGFyID0gIm11Tm9uZSIpCikKCmxvbygKICBkb2N1bWVudGF0aW9uMC53aXRoKCksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya19kb21haW4iKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJ3b3JrX2V4cGVyaWVuY2VfcHJvZ3JhbW1pbmcucyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtfZXhwZXJpZW5jZV9qYXZhLnMiKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJlZHVjYXRpb25fZmllbGQiKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJtbyhlZHVjYXRpb25fbGV2ZWwpIiwgZWRsdmxfcHJpb3IpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtwbGFjZV9wZWVyX3JldmlldyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtwbGFjZV90ZF90cmFja2luZyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoIndvcmtwbGFjZV9wYWlyX3Byb2dyYW1taW5nIiksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya3BsYWNlX2NvZGluZ19zdGFuZGFyZHMiKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJzY2VuYXJpbyIpLAogIGRvY3VtZW50YXRpb24wLndpdGgoImdyb3VwIiksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgib3JkZXIiKQopCmBgYAoKV2UgcGljayBvdXQgc29tZSBpbnRlcmVzdGluZyB2YXJpYWJsZXMgYW5kIHRyeSBjb21iaW5pbmcgdGhlbQoKYGBge3J9CmxvbygKICBkb2N1bWVudGF0aW9uMC53aXRoKCksCiAgZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya19kb21haW4iKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJ3b3JrcGxhY2VfcGVlcl9yZXZpZXciKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKCJ3b3JrcGxhY2VfdGRfdHJhY2tpbmciKSwKICAKICBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtfZG9tYWluIiwgIndvcmtwbGFjZV9wZWVyX3JldmlldyIpKSwKICBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtfZG9tYWluIiwgIndvcmtwbGFjZV90ZF90cmFja2luZyIpKSwKICAKICBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtwbGFjZV9wZWVyX3JldmlldyIsICJ3b3JrcGxhY2VfdGRfdHJhY2tpbmciKSksCiAgCiAgZG9jdW1lbnRhdGlvbjAud2l0aChjKCJ3b3JrX2RvbWFpbiIsICJ3b3JrcGxhY2VfcGVlcl9yZXZpZXciLCAid29ya3BsYWNlX3RkX3RyYWNraW5nIikpCikKYGBgCiMjIENhbmRpZGF0ZSBtb2RlbHMKV2UgaW5zcGVjdCBzb21lIG9mIG91ciB0b3AgcGVyZm9ybWluZyBtb2RlbHMuIAoKQWxsIG1vZGVscyBzZWVtcyB0byBoYXZlIHNhbXBsZWQgbmljZWx5IChyaGF0IGlzIGNhIDEgYW5kIGZsdWZmeSBwbG90cykgdGhleSBhbHNvIGhhdmUgYWJvdXQgdGhlIHNhbWUgZml0IHRvIHRoZSBkYXRhIGVuZCBzaW1pbGFyIGVzdGltYXRlZCBmb3IgdGhlIGhpZ2hfZGVidF92ZXJzaW9uIGJldGEgcGFyYW1ldGVyCgoKYGBge3J9CmRvY3VtZW50YXRpb24wIDwtIGRvY3VtZW50YXRpb24wLndpdGgoKQpzdW1tYXJ5KGRvY3VtZW50YXRpb24wKQpyYW5lZihkb2N1bWVudGF0aW9uMCkKcGxvdChkb2N1bWVudGF0aW9uMCwgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uMCwgbnNhbXBsZXMgPSAxNTAsIHR5cGU9ICJiYXJzIikKCmBgYApgYGB7cn0KZG9jdW1lbnRhdGlvbjEgPC0gZG9jdW1lbnRhdGlvbjAud2l0aCgid29ya3BsYWNlX3BlZXJfcmV2aWV3IikKc3VtbWFyeShkb2N1bWVudGF0aW9uMSkKcmFuZWYoZG9jdW1lbnRhdGlvbjEpCnBsb3QoZG9jdW1lbnRhdGlvbjEsIGFzayA9IEZBTFNFKQoKcHBfY2hlY2soZG9jdW1lbnRhdGlvbjEsIG5zYW1wbGVzID0gMTUwLCB0eXBlPSAiYmFycyIpCgpgYGAKCmBgYHtyfQpkb2N1bWVudGF0aW9uMiA8LSBkb2N1bWVudGF0aW9uMC53aXRoKGMoIndvcmtwbGFjZV9wZWVyX3JldmlldyIsICJ3b3JrX2RvbWFpbiIpKQpzdW1tYXJ5KGRvY3VtZW50YXRpb24yKQpyYW5lZihkb2N1bWVudGF0aW9uMikKcGxvdChkb2N1bWVudGF0aW9uMiwgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uMiwgbnNhbXBsZXMgPSAxNTAsIHR5cGU9ICJiYXJzIikKCmBgYAoKYGBge3J9CmRvY3VtZW50YXRpb24zIDwtIGRvY3VtZW50YXRpb24wLndpdGgoYygid29ya3BsYWNlX3BlZXJfcmV2aWV3IiwgIndvcmtfZG9tYWluIiwgIndvcmtwbGFjZV90ZF90cmFja2luZyIpKQpzdW1tYXJ5KGRvY3VtZW50YXRpb24zKQpyYW5lZihkb2N1bWVudGF0aW9uMykKcGxvdChkb2N1bWVudGF0aW9uMywgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uMywgbnNhbXBsZXMgPSAxNTAsIHR5cGU9ICJiYXJzIikKCmBgYAoKIyMjIEZpbmFsIG1vZGVsCkFsbCBtb2RlbHMgaGF2ZSBzYW1wbGVkIG9rZXkgYnV0IGRvY3VtZW50YXRpb24zIGhhcyBhIGJpdHRlciBmaXQgZm9yIHRoZSBoaWdoIGRlYnQgYmV0YSBwYXJhbWV0ZXIgaXMgc2lnbmlmaWNhbnRseSBiZXR0ZXIgdGhhbiBzb21lIG9mIHRoZSBzaW1wbGUgbW9kZWxzIGFjY29yZGluZyB0byBsb28uIHdlZSBjaG9vc2UgZG9jdW1lbnRhdGlvbjMgYXMgb3V0IGZpbmFsIG1vZGVsLgoKIyMjIFRyeWluZyB0byBhZGQgY292YXJpYW5jZQoKYGBge3J9CmRvY3VtZW50YXRpb24zX3dpdGhfYyA8LSBicm0oCiAgImRvY3VtZW50YXRpb24gfiAxICsgaGlnaF9kZWJ0X3ZlcnNpb24gKyB3b3JrcGxhY2VfcGVlcl9yZXZpZXcgKyB3b3JrX2RvbWFpbiArIHdvcmtwbGFjZV90ZF90cmFja2luZyArICgxIHwgYyB8IHNlc3Npb24pIiwKICBwcmlvciA9IGMoCiAgICBwcmlvcihub3JtYWwoMCwgMC45KSwgY2xhc3MgPSAiYiIpLAogICAgcHJpb3Iobm9ybWFsKDAsIDAuOSksIGNsYXNzID0gIkludGVyY2VwdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdUluY29ycmVjdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdU5vbmUiKSwKICAgIHByaW9yKGxraigyKSwgY2xhc3MgPSAiTCIpCiAgKSwKICBmYW1pbHkgPSBjYXRlZ29yaWNhbCgpLAogIGRhdGEgPSBhcy5kYXRhLmZyYW1lKGQuYm90aF9jb21wbGV0ZWQpLAogIGZpbGUgPSAiZml0cy9kb2N1bWVudGF0aW9uM193aXRoX2MiLAogIGZpbGVfcmVmaXQgPSAib25fY2hhbmdlIgopCgpzdW1tYXJ5KGRvY3VtZW50YXRpb24zX3dpdGhfYykKcmFuZWYoZG9jdW1lbnRhdGlvbjNfd2l0aF9jKQpwbG90KGRvY3VtZW50YXRpb24zX3dpdGhfYywgYXNrID0gRkFMU0UpCgpwcF9jaGVjayhkb2N1bWVudGF0aW9uM193aXRoX2MsIG5zYW1wbGVzID0gMTUwLCB0eXBlPSAiYmFycyIpCmBgYAoKYGBge3J9CmxvbygKICBkb2N1bWVudGF0aW9uMywKICBkb2N1bWVudGF0aW9uM193aXRoX2MKKQpgYGAKQWRkaW5nIHRoZSBjb3ZhcmlhbmNlIHBhcmFtZXRlciBkaWQgbm90IHNpZ25pZmljYW50bHkgaW1wcm92ZSB0aGUgbW9kZWwgYW5kIHdlIHdpbGwgdGhlcmVmb3JlIGNvbnRpbnVlIHRvIHVzZSB0aGUgc2ltcGxlciBtb2RlbCBkb2N1bWVudGF0aW9uMy4KCiMjIyBNb2RlbCB3aXRoIHdpdGggaW5jb21wbGV0ZSBkYXRhIHBvaW50cwoKU29tZSBwYXJ0aWNpcGFudHMgZGlkIG9ubHkgY29tcGxldGUgb25lIHNjZW5hcmlvLiBUaG9zZSBoYXMgYmVlbiBleGNsdWRlZCBmcm9tIHRoZSBpbml0aWFsIGRhdGFzZXQgdG8gaW1wcm92ZSBzYW1wbGluZyBvZiB0aGUgbW9kZWxzLiBXZSBkbyBob3dldmVyIHdhbnQgdG8gdXNlIGFsbCBkYXRhIHdlIGNhbiBhbmQgd2lsbCB0aGVyZWZvcmUgdHJ5IHRvIGZpdCB0aGUgbW9kZWwgd2l0aCB0aGUgY29tcGxldGUgZGF0YXNldC4KCmBgYHtyfQpkb2N1bWVudGF0aW9uMy5hbGwgPC0gYnJtKAogICJkb2N1bWVudGF0aW9uIH4gMSArIGhpZ2hfZGVidF92ZXJzaW9uICsgd29ya3BsYWNlX3BlZXJfcmV2aWV3ICsgd29ya19kb21haW4gKyB3b3JrcGxhY2VfdGRfdHJhY2tpbmcgKyAoMSB8IHNlc3Npb24pIiwKICBwcmlvciA9IGMoCiAgICBwcmlvcihub3JtYWwoMCwgMC45KSwgY2xhc3MgPSAiYiIpLAogICAgcHJpb3Iobm9ybWFsKDAsIDAuOSksIGNsYXNzID0gIkludGVyY2VwdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdUluY29ycmVjdCIpLAogICAgcHJpb3IoZXhwb25lbnRpYWwoMSksIGNsYXNzID0gInNkIiwgZHBhciA9ICJtdU5vbmUiKQogICksCiAgZmFtaWx5ID0gY2F0ZWdvcmljYWwoKSwKICBkYXRhID0gYXMuZGF0YS5mcmFtZShkLmNvbXBsZXRlZCksCiAgZmlsZSA9ICJmaXRzL2RvY3VtZW50YXRpb24zLmFsbCIsCiAgZmlsZV9yZWZpdCA9ICJvbl9jaGFuZ2UiCikKc3VtbWFyeShkb2N1bWVudGF0aW9uMy5hbGwpCnJhbmVmKGRvY3VtZW50YXRpb24zLmFsbCkKcGxvdChkb2N1bWVudGF0aW9uMy5hbGwsIGFzayA9IEZBTFNFKQoKcHBfY2hlY2soZG9jdW1lbnRhdGlvbjMuYWxsLCBuc2FtcGxlcyA9IDE1MCwgdHlwZT0gImJhcnMiKQpgYGAKClRyYWluaW5nIHRoZSBtb2RlbCBvbiBhbGwgZGF0YSBwb2ludHMgcmVkdWNlcyB0aGUgdW5jZXJ0YWludHkgYW5kIGRpZCBub3QgcmVzdWx0IGluIGFueSBzYW1wbGluZyBwcm9ibGVtcy4gV2Ugd2lsbCBwcm9jZWVkIHdpdGggdGhlIG1vZGVsIGZpdHRlZCB0byBhbGwgdGhlIGRhdGEuCgojIyBJbnRlcnByZXRpbmcgdGhlIG1vZGVsCgpFeHRyYWN0IHBvc3RlcmlvciBzYW1wbGVzOgpgYGB7cn0KCnBvc3QgPC0gcG9zdGVyaW9yX3ByZWRpY3QoZG9jdW1lbnRhdGlvbjMuYWxsLCBuZXdkYXRhID0gZGF0YS5mcmFtZSgKICBoaWdoX2RlYnRfdmVyc2lvbiA9IGMoImZhbHNlIiwgInRydWUiKSwKICBzZXNzaW9uID0gTkEsIAogIHdvcmtwbGFjZV9wZWVyX3JldmlldyA9IE5BLCAKICB3b3JrX2RvbWFpbiA9IE5BLCAKICB3b3JrcGxhY2VfdGRfdHJhY2tpbmcgPSBOQQopKQpwb3N0LmxvdyA8LSAgcG9zdFssMV0KcG9zdC5oaWdoIDwtIHBvc3RbLDJdCnN1bW1hcnkocG9zdCkKCmBgYAo=